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I.  INTRODUCTION 

This  Research  Note,  Utilization  of  Tactical  Computers  for  Training: 

Field  Evaluation  Plan,  describes  Phase  V  in  the  development  of  TACFIRE 
Automated  Instruction  (Al)  courseware.  Figure  1  depicts  the  five  phases 
that  constitute  the  total  project  effort. 

The  overall  project  aim  Is  to  extend  the  scope  of  the  application  of  computer- 
assisted  interaction  (CAI)  to  the  development  of  self-instructive  programs  and 
procedures  for  users  of  tactical  data  processing  systems.  The  basic  approach 
is  to  provide  AI  training  subsystem  packages  which  can  be  run  on  the  operating 
system  and,  when  not  used  for  tactical  operations,  to  provide  initial  and 
refresher  training  in  system  use.  The  overall  objective  of  this  work  effort 
is  the  development  of  stand  alone  CAI  courseware  appropriate  to  the  training 
of  users  of  the  TACFIRE  system.  Project  products  will  provide  the  foundation 
for  subsequent  evaluation  and  refinement  of  CAI  technology  as  applied  to 
training  in  tactical  systems. 

The  purpose  of  Phase  V  is  to  develop  an  evaluation  plan  for  demonstrating  the 
execution  and  effectiveness  of  the  entire  TACFIRE  AI  courseware  package.  The 
evaluation  plan  covers:  (1)  procedures  and  methodology  for  performing  a  review 
of  the  TACFIRE  courseware  content  by  subject  matter  experts  at  the  U.S.  Army 
Field  Artillery  School,  Fort  Sill,  Oklahoma;  (2)  procedures  and  requirements 
for  demonstrating  the  execution  of  TACFIRE  courseware  on  the  ARI  and  TACFIRE 
operating  systems;  and  (3)  procedures  for  assessing  the  acceptability  of 
TACFIRE  AI  courseware  by  field  artillery  personnel. 

The  evaluation  plan  will  be  reviewed  by  the  U.S.  Army  Research  Institute  (ARI) 
in  conjunction  with  the  U.S.  Army  Field  Artillery  School.  Implementation  of  the 
evaluation  plan  and  the  formative  evaluation  of  the  instructional  effectiveness 
of  the  learning  materials  will  be  the  responsibility  of  ARI.  Prior  to  the  imple¬ 
mentation  of  the  evaluation  plan,  the  courseware  will  have  been  debugged  so  that 
it  executes  properly  on  the  AN/GYK-12  computer. 
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Phase  I  -  Analyze  System  and  Training  Requirements 
Phase  II  -  Perform  Job/Task  and  Training  Analysis 
Phase  III  -  Develop  Courseware 
Phase  IV  -  Install  Courseware 


II.  REVIEW  OF  TACFIRE  AI  COURSEWARE 


TACFIRE  AI  courseware  is  an  operational  training  program  with  an  expected  use 
in  TACFIRE  field  units  and  at  the  U.S.  Army  Field  Artillery  School  (USAFAS) . 

The  review  process  has  the  purpose  that  the  training  analysis  products  (criterion 
and  enabling  objectives,  test  items,  etc.)  and  courseware  accurately  reflect 
TACFIRE  operations,  procedures,  content,  doctrine,  and  tactical  use  as  employed 
on  the  job  in  the  field  situation.  The  review  process  also  ensures  that  the 
courseware  executes  properly  on  the  TACFIRE  equipment.  The  U.S.  Army  Field 
Artillery  School  (USAFAS)  has  TACFIRE  subject  matter  experts  and  operational 
TACFIRE  equipment  at  the  persent  time.  It  is  expected  that  the  review  of  the 
TACFIRE  AI  courseware  will  take  place  at  USAFAS,  Fort  Sill,  Oklahoma. 

The  review  process  is  expected  to  follow  the  training  development  process  to 
logically  proceed  from  course  objectives  to  course  content  to  execution  of 
courseware. 

A.  Review  of  Course  Objectives 

The  training  tasks,  criterion  and  enabling  objectives,  and  test  items  to  be 
included  in  the  course  are  defined  in  the  Phase  II  report,  "Utilization  of 
Tactical  Computers  for  Training:  Job  Task  Training  Analysis,"  TM-5544/001/00, 
dated  20  August  1975. 

This  portion  of  the  review  is  to  examine  the  job/task  training  analysis 
(Phase  II)  report  and  indicate  modifications  to  the  criterion  and  enabling 
objectives  and/or  accompanying  test  items. 

The  number  of  reviewers  required  should  be  determined  by  the  USAFAS.  Even  for 
subject  matter  experts,  there  are  certain  areas  an  individual  knows  very  well 
but  other  areas  are  better  known  by  other  subject  matter  experts.  Also,  the 
amount  of  material  to  be  reviewed  lends  itself  to  different  subject  matter 
experts  reviewing  different  areas. 
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The  actual  procedure  recommended  for  carrying  out  the  review  is  to  use  copies 
of  the  Phase  II  report  to  note  discrepancies  and  corrections.  The  notation 
should  not  only  identify  (mark)  the  problem  area  but  also  indicate  why  it  is 
wrong.  A  separate  list  of  the  pages  on  which  these  occur  would  ensure  that 
none  are  overlooked.  It  it  suggested  that  one  of  the  Phase  II  reports  be  used 
as  a  "master"  copy  for  all  changes  which,  in  turn,  will  be  used  to  identify 
where  changes  should  be  made  in  the  course  content.  The  audit  trail  used 
pinpoints  where  in  the  course  content  the  change  is  to  be  made.  The  intent 
of  this  procedure  is  to  minimize  the  amount  of  paperwork  and  number  of 
documents  required  to  identify,  specify,  and  record  what  needs  to  be  changed. 
The  report  itself  can  be  easily  used  for  this  purpose. 

B.  Review  of  Course  Content 

This  review  is  concerned  with  the  accuracy  of  the  courseware  and  consists  of 
an  off-line  (off  computer)  check  of  the  accuracy  of  the  course  content  using 
the  computerized  listing  (printouts)  of  the  course. 

Listings  (printouts  of  course  card  decks)  provide  the  entire  course  content 
including  the  instructional  text,  test  items,  answer  processing,  feedback, 
branching,  remedial  instruction,  and  decisions  made  (decision  frames)  which 
determine  the  student's  progress  through  the  course. 

The  listings  provide  an  easy  method  of  reviewing  course  content  and  identifying 
and  recording  problem  areas.  The  subject  matter  experts  should  be  familiar 
with  the  PLANIT  symbology  (B:)  used  for  branching  in  order  to  tie  together  the 
frame  sequences  including  remedial  instruction.  Knowing  how  to  interpret 
decision  (D)  frames  would  also  be  helpful  in  determining  the  adequacy  of 
the  pass-fail  levels  for  the  tests  at  the  end  of  each  lesson  and  module. 


The  procedure  recommended  is  to  go  through  the  course  with  the  off-line  course 
exhibits  and  identify  on  the  listing  and/or  exhibits  discrepancies  and 
questionable  areas  which  may  exist  along  with  notations  which  specify  why  it 
is  a  discrepancy.  The  listing  thus  provides  a  record  of  the  discrepancies, 
minimizing  the  paper  work  and  record  keeping  required.  One  copy  of  the  listings 
and  off-line  course  exhibits  should  be  a  "master”  copy  to  show  all  changes 
which  should  be  effected.  These  changes  can  then  be  made  (edited)  easily 
on-line  to  update  the  course  content. 

C .  Review  of  Execution  of  TACFIRE  Courseware 

The  TACFIRE  AI  courseware  needs  to  be  run  on  line  to  ensure  that  the  course¬ 
ware  can  be  run  (executed)  on  the  TACFIRE  computer  system  and  that  the  frame 
size  including  feedback  and  the  following  frames  fit  the  configuration  of  the 
TACFIRE  CRT  display  (C/ED)  and  programming  parameters.  The  content  should  be 
also  checked  as  the  individual  frame-by-frame  presentation  on  the  CRT  may 
appear  different  than  on  the  listing,  as  there  is  no  opportunity  to  refer  to 
a  number  of  frames  at  the  same  time  on  the  C/ED.  This  is  particularly  true  of 
test  items  which  should  be  examined  closely  to  determine  that  enough  of  the 
situational  context  of  the  question  is  given  to  adequately  respond  to  the  test 
question. 

The  procedure  is  simple.  A  reasonable  check  can  be  made  by  going  through  the 
entire  course,  once  with  all  correct  responses  and  a  second  time  with  all 
incorrect  responses.  The  subject  matter  experts  should  take  the  course  in 
the  "author"  mode  which  will  permit  them  to  continue  if  they  encounter 
problems  within  the  on-line  material.  Each  examines  the  content  of  the  course 
and  uses  the  listing  to  provide  the  correct  or  incorrect  response  and  to  record 
where  problems  occur  and  what  the  problem  is,  including  possible  feedback 
from  the  PLANIT  system.  This  procedure  is  facilitated  by  having  the  Electronic 
Line  Printer  (ELP)  in  the  "on"  status.  This  will  provide  a  record  of  the 
actions  (responses)  taken  while  running  the  course.  At  the  end  of  the  run, 
PLANIT  student  records  should  be  obtained. 
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The  problem  areas  that  are  identified  (content  and  execution)  need  to  be 
analyzed  to  determine  the  corrective  action  to  be  taken.  Changes  to  the 
courseware  are  easily  made  by  on-line  editing.  These  changes  are  recorded  on 
the  listing  to  provide  an  updated  record  of  the  course  content. 
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III.  TRAINEE  ACCEPTABILITY  OF  TACFIRE  A1  COURSEWARE 

Subject  matter  experts  provide  feedback  on  content  validity  and  organization  of 
the  course  and  the  fact  that  the  courseware  executes  (runs)  properly  on  the 
TACFIRE  system.  TACFIRE  trainees  provide  feedback  on  levels  of  understanding 
and  mastery  and  interactive  computer  behavior,  verifying  that  the  courseware 
executes  and  is  acceptable. 

Having  TACFIRE  trainees  take  the  TACFIRE  AI  course  provides  answers  to  two  basic 
questions:  (1)  Do  they  learn  the  TACFIRE  operations  specified  in  the  objectives?; 
and  (2)  What  attitude  do  they  have  toward  the  TACFIRE  AI  course?  The  first 
question  is  answered  by  analyzing  trainee  results  on  the  criterion  tests  cover¬ 
ing  the  objectives.  The  second  question  is  answered  by  obt lining  trainee 
reaction  to  the  learning  process  by  a  structured,  semi  open-ended  questionnaire 
upon  completion  of  the  course.  Implicit  in  both  questions  is  whether  the 
TACFIRE  AI  course  is  "GI  proof.”  Experience  with  trainees  during  the  evaluation 
study  should  provide  some  indication  of  this  as  will  earlier  runs  using  military 
personnel  for  course  checkout. 

A.  Trainee  Selection  Requirements  for  the  Evaluation  Study 

The  TACFIRE  AI  course  is  intended  to  train  field  artillery  personnel  in  TACFIRE 
Battalion  Fire  Direction  Center  operations.  Trainees  selected  for  the  evalua¬ 
tion  process  should  be  representative  of  the  pool  of  field  artillery  personnel 
who  would  be  selected  for  TACFIRE  FDC  training.  Only  in  this  way  can  the  re¬ 
sults  obtained  be  easily  generalized  to  the  larger  group.  This  also  applies 
to  the  prerequisites  for  selection  to  TACFIRE  training.  A  number  of  these 
factors  for  selecting  trainees  for  the  evaluation  study  are  considered  in  the 
paragraphs  which  follow. 

1.  Field  Artillery  Skills  and  knowledge 

The  primary  positions  in  the  TACFIRE  Battalion  FDC  operations  are  the  Fire  Di¬ 
rection  officer  (Captain),  MOS  01193  and  the  Fire  Direction  Sergeant  (E6) ,  MOS 
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13E30.  Each  of  these  positions  is  expected  to  be  filled  by  experienced  field 
artillery  personnel. 

The  prerequisites  required  for  the  Fire  Direction  Sergeant  are:  E4  or  above, 
qualified  in  MOS  13E,  MOS  evaluation  score  of  100  or  more  on  the  latest  Enlisted 
Evaluation  Data  Report.  For  the  Fire  Direction  officer,  the  prerequisites  are: 
a  graduate  of  the  Field  Artillery  Officer  Basic  Course  and  have  18  months  of 
active  duty  remaining  at  the  completion  of  the  course. 

Selection  Requirement:  Personnel  selected  for  the  evaluation  study  should 
reasonably  meet  the  above  criteria.  Personnel  files  of  those  being  considered 
for  the  study  should  be  checked  to  verify  that  they  do. 

Potential  Assignment  to  TACFIRE.  The  TACFIRE  AI  course  is  intended  to  train 
TACFIRE  personnel.  Those  in  the  personnel  pool  for  selection  for  TACFIRE 
training  have  a  reasonable  expectation  that,  if  selected,  they  will  be 
assigned  to  TACFIRE  or  need  to  know  TACFIRE  operations.  Consequently, 
they  are  motivated  to  learn  TACFIRE  operations. 

Personnel  selected  for  the  evaluation  should  also  have  a  reasonable  expectation 
that  they  will  need  to  know  TACFIRE  operations  and,  consequently,  have  the  same 
level  of  motivation  to  learn.  Note  that  this  does  not  imply  that  there  is  a 
high  level  of  motivation  but  simply  that  the  skills  and  knowledges  learned  will 
be  of  value  to  the  trainee.  Personnel  who  are  awaiting  discharge,  reassignment, 
or  are  presently  assigned  to  areas  other  than  field  artillery  operations 
should  not  be  selected  to  participate  in  the  evaluation. 

Selection  requirement:  Current  or  potential  assignment  to  TACFIRE  operations 
or  a  duty  position  requiring  TACFIRE  skills  and  knowledges. 
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Reading  and  Visual  Skills.  TACFIRE  FDC  operations  personnel  need  to 
see,  read,  and  interpret  various  displays  and  devices  (RD,  CED,  ELP,  DPM) 
and  have  the  verbal  skills  to  understand  and  communicate  with  field  artillery 
personnel  on  field  artillery  operations.  A  minimum  reading  level  is  required 
as  well  as  no  disqualifying  visual  handicaps. 

Selection  Requirement:  Minimum  ACB  GT  score  of  90  or  above  (8th  grade)  and 
no  disqualifying  visual  handicap  on  Physical  Status  Profile  (PULHES). 

When  levying  personnel  requirements  for  an  evaluation  study,  the  normal  proce¬ 
dure  for  the  organization  being  levied  upon  is  to  send  whatever  personneol  are 
available  at  the  time  that  will  fill  the  requirement.  This  often  results  in 
a  "warm  body"  syndrome  that  may  give  atypical  personnel.  A  better  procedure 
is  to  analyze  the  personnel  pool  beforehand  and  provide  the  supplying  organi¬ 
zation  with  a  list  of  qualified  personnel  from  which  to  select  a  certain 
number  for  the  period  specified.  This  provides  flexibility  for  the  supplying 
organization  and  ensures  obtaining  personnel  representative  of  the  TACFIRE 
selection  pool. 

Number  of  Trainees  Required  for  the  Evaluation  Study.  Careful  controlled 
preselection  of  trainee  personnel  to  ensure  they  are  typical  candidates 
for  TACFIRE  training  reduces  the  number  required  for  the  evalution  study. 
Nonqualifying  individuals  are  eliminated  from  the  study  as  are  those  with 
reading  or  visual  handicaps  and  those  with  no  inherent  interest  in  TACFIRE 
training. 

If  the  course  is  effective,  scores  on  the  criterion  tests  should  cluster  at 
the  high  end  of  the  scale.  If  the  course  is  not  effective,  scores  should 
cluster  at  the  low  end  of  the  scale.  This  should  be  a  fairly  constant  factor, 
whether  a  large  number  or  small  number  of  trainees  are  used. 


However,  no  matter  how  good  preselection  procedures  are,  there  is  the  occasional 
"maverick"  who  doesn't  like  any  kind  of  training,  won't  learn  or  can't  learn  be¬ 
cause  of  personal  problems  or  bias.  There  is  also  the  "maverick"  who  seems  to 
learn  no  matter  how  or  what  material  is  presented.  Any  one  individual  has  an 
equal  probability  of  being  selected  as  any  other  one  individual  in  the  selection 
pool.  This  applies  to  both  the  high  and  low  ends  of  the  spectrum.  The  number 
of  trainees  selected  for  the  evaluation  study  should  be  large  enough  to  easily 
identify  (isolate)  the  occasional  individual  at  one  or  the  other  extreme  and 
still  indicate  what  group  performance  will  be. 


Another  consideration  is  that,  at  present,  there  is  little  data  on  measured 
performance  relative  to  pass-fail  (go-no  go)  scores.  Usual  methods  of  deter¬ 
mining  pass-fail  scores  may  be  inappropriate.  Prior  to  the  evaluation  study, 
each  functional  area  module  test  will  have  been  reviewed  by  subject  matter 
experts  and  considered  valid  measures  of  performance  for  the  stated  objectives. 
In  considering  pass-fail  scores,  an  upper  limit  of  test  scores  for  trainees 
could  be  those  test  scores  obtained  by  subject  matter  experts  and  instructors. 
However,  setting  a  pass-fail  score  in  this  fashion  may  be  too  low.  Previous 
research  (Navy  Computer  Integrated  Instruction  [CII])  indicates  that  it  is 
not  uncommon  for  subject  matter  experts  (instructors)  as  a  group  to  have  mean 
scores  of  74  to  89%  (median  of  76%)  on  various  test  modules  that  comprehen¬ 
sively  cover  the  functional  areas  being  taught.  Another  criteria  for  pass-fail 
scores,  used  for  many  military  courses,  is  a  score  of  70  or  73%.  For  MOS  pro¬ 
ficiency  test  scores,  in  some  cases,  the  passing  score  is  fairly  close  to  the 
chance  score  that  could  be  obtained  by  randomly  answering  all  the  multiple 
choice  questions.  Both  70%  or  subject  expert  scores  may  be  too  low.  The 
number  of  trainees  selected  for  the  evaluation  study  should  be  therefore  large 
enough  to  provide  a  reasonable  measure  of  performance  expected  (pass-fail)  as 
a  result  of  taking  the  course. 

Another  consideration  is  to  determine  the  internal  consistency  of  the  course. 

If  the  errors  made  by  trainees  are  randomly  distributed  over  performance 
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objectives,  the  modules  can  be  considered  satisfactory.  However,  if  a  number 
of  trainees  fail  particular  test  items  covering  specific  objectives,  then  the 
training  given  is  suspect  and  should  be  revised.  The  number  of  trainees  should 
be  lar0  enough  so  that  poor  performance  areas  can  be  isolated.  The  audit  trail 
readily  links  together  course  content,  objectives,  and  test  items. 

Based  upon  the  above  considerations,  the  number  of  trainees  recommended  for 
the  evaluation  study  is  ten  (10) ,  all  of  whom  meet  the  prerequisites  for  TAC¬ 
FIRE  training.  This  will  provide  for  significant  clusters  of  scores  on  the 
module  tests  and  easily  isolate  individuals  with  atypical  performance.  It 
will  provide  objective  data  to  tentatively  establish  pass-fail  (go  no-go) 
scores,  provide  a  measure  of  the  internal  consistency  of  the  course  content, 
and  the  range  of  training  time  required  to  take  the  course.  It  is  also  suffi- 
,  ciently  large  to  provide  some  measure  of  confidence  that  the  results  obtained 

will  apply  to  the  larger  pool  of  TACFIRE  trainees. 

C.  Equipment,  Support  Requirements  and  Schedule 

1.  Equipment 

The  TACFIRE  AI  course  is  designed  to  run  on  the  TACFIRE  Fire  Direction  System. 

It  is  expected  that  the  evaluation  study  will  take  place  on  the  TACFIRE  system 
at  the  U.S.  Army  Field  Artillery  School,  Fort  Sill,  Oklahoma.  Trainees  taking 
the  course  will  use  one  of  three  types  of  devices  available:  (1)  the  ACC  con- 
sole,  (2)  the  VFMED,  or  (3)  the  modified  MIOD.  It  is  expected  that  the  avail¬ 
ability  of  equipment  will  provide  a  distribution  of  equipment  users,  e.g.,  some 
on  ACC,  some  on  VFMED  and  MIOD,  and  some  on  a  combination  of  ACC  and  VFMED /MIOD . 

2.  Support  Requirements 

Personnel  will  be  required  for  the  operation  of  the  TACFIRE  tactical  equipment. 
Support  personnel  for  the  evaluation  study  are  dependent  on  the  number  of 
trainees  taking  the  AI  course  at  any  given  time.  A  minimum  of  two  evaluation 
personnel  is  recommended.  Their  functions  fall  into  two  general  areas:  (1) 
load  the  PLAN IT  system  and  AI  courseware  onto  the  TACFIRE  system  and  obtain 
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the  PLANIT  trainee  records  at  the  end  of  each  session;  and  (2)  conduct  and 
monitor  the  evaluation  study  including  briefing  the  trainees  at  the  beginning 
of  the  course,  completing  the  forms  required  and  Interviewing  each  trainee  at 
the  end  of  the  course.  The  TACFIRE  AI  course  runs  by  itself  and  trainees  work 
at  their  own  pace.  Course  completion  times  will  differ  and  the  end  of  course 
interviews  conducted  will  be  spread  over  a  period  of  time,  necessitating  a  mini¬ 
mum  number  of  support  personnel.  All  course  materials  and  data  gathering  forms 
are  provided.  Other  than  the  operational  equipment  and  ELP  printouts  of 
student  records,  the  additional  support  requirements  anticipated  are  two 
tape  recorders,  10  tape  cassettes  to  tape  the  interviews,  and  table  and  chairs 
in  a  quiet  place  to  conduct  the  interviews. 


3 .  Schedule 

A  two-week  maximum  liability  period  for  each  trainee  Is  recommended.  This 
should  allow  the  slowest  trainees,  working  at  their  own  pace,  to  finish  the 
course.  The  TACFIRE  AI  course,  still  under  development,  is  currently  esti¬ 
mated  to  take  approximately  25  hours  of  on-line  Instruction  on  the  ACC.  The 
other  two  devices  will  require  a  somewhat  longer  time  period  because  of  the 
additional  steps  required  for  the  trainee  to  enter  his  response.  Assuming 
the  25  hours  is  reasonable,  trainee  times  (working  at  their  own  individual 
pace)  will  probably  range  from  20  to  40  hours,  with  the  median  times  around 
25  hours. 

Considering  the  initial  briefing,  forms  to  be  filled  out,  the  interview, 
possible  TACFIRE  system  down  time,  the  normal  vagaries  of  other  military  and 
personal  requirements,  the  two-week  maximum  liability  period  seems  reasonable 
and  adequate  for  planning  purposes  at  this  time.  After  the  TACFIRE  AI  course¬ 
ware  has  bean  developed  and  run  on  the  TACFIRE  system,  the  estimated  time  re¬ 
quired  can  be  verified. 

The  availability  and  status  of  the  equipment  consistent  with  other  requirements 
at  the  Field  Artillery  School  will  determine  when  trainees  can  get  on  the  TACFIRE 
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system  and  also  the  number  of  trainees  who  can  take  the  course  at  any  one 
time.  As  far  as  the  evaluation  study  is  concerned,  trainees  can  take  the 
course  individually  or  all  at  the  same  time.  A  rigid  time  schedule  is  not 
required  since  PLANIT  has  a  complete  record-keeping  capability.  The  trainee 
can  sign  off  at  any  time  and  PLANIT  will  start  him  at  his  place  in  the  course 
when  he  returns.  There  are  "natural"  places  to  end  a  day's  session,  as  at 
the  end  of  a  lesson  or  end  of  a  module.  Trainees  can  be  expected  to  leave 
at  different  times. 

Procedures 


Initial  Activities.  Trainees  will  report  to  the  monitor,  fill  out  the 
introductory  form  (Figure  2),  and  be  briefed  onthe  TACFIRE  AI  course.  The 
monitor  establishes  a  file  (individual  folder)  for  each  trainee  which  also 
contains  the  TACFIRE  Course  Data  Sheet  (Figure  3). 

Taking  the  TACFIRE  AI  Course.  After  assignment  to  a  console,  trainees 
follow  the  printed  instructions  (Figure  4)  and  log  in  with  their  trainee 
ID  number.  They  then  take  the  course.  For  lunch  and  other  breaks  and  at  the 
end  of  each  daily  session,  students  log  off  and  then  log  in  again  when 
they  return  to  resume.  Students  are  free  to  take  coffee  or  latrine  breaks 
whenever  they  so  desire  during  each  session.  Trainee  activities  are 
monitored  and  also  logged  on  and  off  manually  by  the  group  monitor.  System 
downtimes  are  also  recorded  by  the  monitor.  Trainees  continue  to  take  the 
course  until  completion  including  the  module  tests  which  are  given  on-line 
as  part  of  the  course. 


Interview.  At  the  time  he  completes  the  TACFIRE  AI  course,  each  trainee  is 
interviewed  in  depth  in  regard  to  their  experience  with  TACFIRE  AI. 


NAME 

TACFIRE  COURSE  DATA  QUESTIONNAIRE 

DATE 

SSAN 

GRADE 

TIME  IN  GRADE 

MOS 

JOB  TITLE 

UNIT 

PHONE  NUMBER 

DATE  OF  BIRTH 

LENGTH  OF  SERVICE 

ETS  DATE 

FIELD  ARTILLERY  EXPERIENCE: 

1.  DATT:  FROM 

TO 

GRADE 

MOS 

DUTY  POSITION  _ 

LOCATION 

2.  DATE:  FROM 

TO 

GRADE 

MOS 

DUTY  POSITION 

LOCATION  _ 

Figure  2.  Introductory  Form. 


TACFIRE  COURSE  DATA  SHEET 


ID  NUMBER:  1 

NAME  _ 

MOS  _ 

ORGANIZATION  _ 
PHONE  NUMBER  _ 
GT  SCORE 


GRADE 


JOB  TITLE 


LATEST  MOS  EVALUATION  SCORE 
ETS  DATE 


EDUCATION  _ 

(Grade  Completed  or  Degree) 

ION  SCORE  MOS 


DATE  OF  BIRTH  _ 

LENGTH  OF  SERVICE 


MODULE  1. 

CONSOLE  ASSIGNED: 
START  DATE  AND  TIME 

ON  CONSOLE  TIME 


VFMED  _  MIOD  _ 

END  DATE  AND  TIME 


DAY  1:  1.  FROM 

TO 

2.  FROM 

TO 

£ 

3.  FROM 

TO 

4.  FROM 

TO 

——  n 

■% 

DAY  2:  1.  FROM 

TO 

2.  FROM 

TO 

* 

3.  FROM 

TO 

4.  FROM 

TO 

H 

DAY  3:  1.  FROM 

TO 

2.  FROM 

TO 

i 

3.  FROM 

TO 

4.  FROM 

TO 

SYSTEM  DOWN  TIME 

1.  DATE 

FROM 

TO 

2.  DATE 

FROM 

n  1 

TO  1  i 

3.  DATE 

FROM 

TO 

4.  DATE 

FROM 

TO  1  i 

TOTAL  TIME  ON  CONSOLE 

FIRST  TEST  SCORE  _ 

SECOND  TEST  SCORE 


TIME  REQUIRED 
TIME  REQUIRED 


Figure  3.  TACFIRE  Course  Data  Sheet. 
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MODULE  2. 

' 

CONSOLE  ASSIGNED:  ACC 

VFMED 

MIOD 

START  DATE  AND  TIME 

END  DATE  AND  TIME 

i 

ON  CONSOLE  TIME 

DAY  1:1.  FROM 

TO 

2.  FROM 

TO 

3.  FROM 

TO 

4.  FROM 

TO 

*  DAY  2:  1.  FROM 

TO 

2.  FROM 

TO 

tc 

3.  FROM 

TO 

4.  FROM 

TO 

DAY  3:  1.  FROM 

TO 

2.  FROM 

TO 

i 

3.  FROM 

TO 

4.  FROM 

TO 

£ 

SYSTEM  DOWN  TIME 

•«» 

■■ 

1 .  DATE  FROM 

TO 

2.  DATE 

FROM 

TO 

3.  DATE  FROM 

TO 

4.  DATE 

FROM 

TO 

|  f  TOTAL  TIME  ON  CONSOLE 

i  <  FIRST  TEST  SCORE 

TIME  REQUIRED 

|  SECOND  TEST  SCORE 

TIME  REQUIRED 

1 

MODULE  3. 

V 

CONSOLE  ASSIGNED:  ACC 

VFMED 

MIOD 

1 

START  DATE  AND  TIME 

END  DATE  AND  TIME 

ON  CONSOLE  TIME 

3 

DAY  1:  1.  FROM 

TO 

2.  FROM 

TO 

f 

3.  FROM 

TO 

4.  FROM 

TO 

t 

DAY  2:  1.  FROM 

TO 

2.  FROM 

TO 

3.  FROM 

TO 

4.  FROM 

TO  ' 

i 

DAY  3:  1.  FROM 

TO 

2.  FROM 

TO 

| 

» 

3.  FROM 

TO 

4.  FROM 

TO 

1 

SYSTEM  DOWN  TIME 

1 .  DATE  FROM 

TO 

2.  DATE 

FROM 

TO 

1 

3.  DATE  FROM 

TO 

4.  DATE  “ 

FROM 

TO 

I 

i 

i 

v 

TOTAL  TIME  ON  CONSOLE 

FIRST  TEST  SCORE 

TIME  REQUIRED 

V  SECOND  TEST  SCORE 

TIME  REQUIRED 

N 

1 

Figure  3. 

TACFIRE  Course  Data  Sheet 

(Cont'd) . 
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MODULE  4. 

CONSOLE  ASSIGNED:  ACC  _  VFMED  _  MIOD 

START  DATE  AND  TIME  _  END  DATE  AND  TIME 

ON  CONSOLE  TIME 


DAY  1:  1.  FROM 

TO 

2. 

FROM 

TO 

3.  FROM 

TO 

"  4. 

FROM 

TO 

DAY  2.  1.  FROM 

TO 

2. 

FROM 

TO 

3.  FROM 

TO 

4. 

FROM 

TO 

DAY  3:  1.  FROM 

TO 

2. 

FROM 

TO 

3.  FROM 

TO 

4. 

FROM 

TO 

SYSTEM  DOWN  TIME 

1.  DATE 

FROM 

TO 

2. 

DATE 

FROM 

TO 

3.  DATE 

FROM  _ 

TO 

4. 

DATE 

FROM 

TO 

TOTAL  TIME  ON  CONSOLE  _ 

FIRST  TEST  SCORE  _  TIME  REQUIRED 

SECOND  TEST  SCORE  _  TIME  REQUIRED 


MODULE  5. 

CONSOLE  ASSIGNED:  ACC 

VFMED 

MIOD 

START  DATE  AND  TIME 

END  DATE  AND  TIME 

ON  CONSOLE  TIME 

DAY  1:  1.  FROM 

TO 

2.  FROM 

TO 

3.  FROM 

TO 

4.  FROM 

TO 

DAY  2:  1.  FROM 

TO 

2.  FROM 

TO 

3.  FROM 

TO 

4.  FROM 

TO 

DAY  3:  1.  FROM 

TO 

2.  FROM 

TO 

3.  FROM 

TO 

4.  FROM 

TO 

SYSTEM  DOWN  TIME 

1 .  DATE  FROM 

TO 

2.  DATE 

FROM 

3.  DATE  FROM 

TO  " 

4.  DATE 

FROM 

TOTAL  TIME  ON  CONSOLE 

FIRST  TEST  SCORE 

TIME  REQUIRED 

SECOND  TEST  SCORE 

TIME  REQUIRED 

Figure  3.  TACFIRE  Course  Data  Sheet  (Cont'd). 
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SUMMARY 
MODULE  1: 

TIME  REQUIRED  _  TEST  SCORE  TEST  TIME 

CONSOLE:  ACC  _  VFMED  _ MIOD  _ 

MODULE  2: 

TIME  REQUIRED  _  TEST  SCORE  TEST  TIME 

CONSOLE:  ACC  _  VFMED  _  MIOD  _ 

MODULE  3: 

TIME  REQUIRED  _  TEST  SCORE  TEST  TIME 

CONSOLE:  ACC  _  VFMED  _  MIOD  _ 

MODULE  4: 

TIME  REQUIRED  _  TEST  SCORE  TEST  TIME 

CONSOLE:  ACC  _  VFMED  _  MIOD  _ 

MODULE  5: 

TIME  REQUIRED  _  TEST  SCORE  TEST  TIME 

CONSOLE:  ACC  _  VFMED  _ _  MIOD  _ 


INTERVIEW:  DATE  _  START  TIME  _  END  TIME 

INTERVIEWER  ___________________  TAPED:  YES  _  NO 


Figure  3.  TACFIRE  Course  Data  Sheet  (Cont'd). 
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When  you  see 

Type  (Exactly  as  spaced)  and  Enter 

LOG  IN  OR  END 

★ 

Your  I.D.  -  example  H2304163, 

ENTER  COMMAND 

★ 

GET  FM1 

IDENTIFY  YOURSELF 

* 

Your  I.D.  -  example  H23041 63 

When  you  take  a  break 

FINISHED 

Figure  4. 


Trainee  Log  On  and  Log  Off  Instructions 


The  interviews  are  conducted  using  the  TACFIRE  AI  Debriefing  Questionnaire 
(Appendix  A)  as  a  basis.  The  trainee  is  asked  whether  he  has  any  objection 
to  the  interview  being  recorded  (taped).  If  he  does  not,  the  interview  is 
recorded  using  the  tape  cassette  recorder.  Other  checks  on  the  Privacy  Act 
of  1974  restraints  should  also  be  made.  The  interviewer  fills  out  the 
interview  form  based  upon  the  trainee's  responses.  To  assist  in  this  process, 
the  trainee  is  given  a  copy  of  the  blank  interview  form.  Some  questions  are 
open-ended  and  others  require  a  specific  response.  Trainees  are  encouraged 
to  amplify  their  responses. 

The  interview  should  take  place  in  a  quiet  location  where  there  will  be 
little  disruption  in  terms  of  background  noise  or  curious  onlookers. 

End  Activities.  The  monitor  should  obtain  the  PLANIT  trainee  records 
and  place  them  in  the  individual  file  along  with  the  other  records  and 
forms  completed,  including  the  record  or  system  down  time  for  each  trainee. 
These  are  used  for  the  analyses  which  follows.  Each  t.j.nee  fi  t  should 
contain  the  following: 

1.  TACFIRE  Introductory  Form 

2.  TACFIRE  Course  Data  Sheet 

3.  Complete  PLANIT  Trainee  Performance  Records  for  all  modules 
including  test  scores  and  on-line  times 

4.  The  completed  AI  Debriefing  Questionnaire  and  Tape  Cassette  Record 
of  the  interview 

5.  Comments  of  the  monitor. 

In  addition,  the  monitor  should  secure  the  off-line  materials  (exhibits) 
and  module  test  exhibits. 
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ANALYSIS  OF  THE  DATA 


The  two  questions  to  be  answered  in  the  anlaysis  are:  (a)  Do  trainees  learn 
the  TACFIRE  operations  specified  in  the  objectives?,  and  (b)  What  attitude 
do  the  trainees  have  toward  the  TACFIRE  AI  courseware?  Answers  to  the  first 
question  are  obtained  by  analyzing  the  PLANIT  trainee  records.  Answers  to 
the  second  question  are  obtained  by  analyzing  the  trainee  resposes  to  the 
TACFIRE  AI  Debriefing  Questionnaire. 

Do  Trainees  Learn?  Two  sets  of  date  with  summary  statistics  should  be 
compiled.  One  for  trainees  characteristics,  the  other  for  trainee  performance 

Trainee  Characteristics.  A  summary  ssheet  should  be  prepared  showing  the 
data  for  each  trainee  on  the  TACFIRE  course  data  sheet  (Figure  5).  Frequency 
distributions  of  each  variable  should  be  obtained  as  well  as  the  mean, 
standard  deviation,  and  rai>  e  of  values  for  the  variable,  GT  score,  mean  = 
104.5,  standard  deviation  of  7.5,  range  96  to  125.  These  are  compiled  to 
(a)  determine  that  the  selection  criteria  have  been  met,  (b)  allow  later 
comparison  against  course  results,  and  (c)  determine  the  distribution  of 
characteristics  in  the  trainee  sample. 


Trainee  Number 

GRADE 

MOS 

AGE 

GT  Score 

Education 

1 

E6 

13E30 

27 

108 

12 

2 

E4 

13E20 

22 

117 

14 

3 

E6 

13E30 

26 

102 

12 

Figure  5.  Summary  of  Trainee  Characteristics 
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Trainee  Performance.  A  summary  sheet  should  be  prepared  showing  the  course 
data  for  each  trainee  time  required  for  each  module,  module  test  score, 
total  time  required  for  the  course,  and  average  of  his  module  test  scores. 
Frequency  distributions  of  each  variable  should  be  obtained  as  well  as  the 
means,  standard  deviation  and  range  of  values  for  the  variable,  FM  Module, 
Module  time,  mean  =  11.2  hours,  standard  deviation  of  1.6,  range  9.5  to 
14.2  hours. 

To  answer  the  question  "Do  trainees  learn?"  the  frequency  distributions  of 
test  scores  for  each  module  is  examined  (Figure  6).  For  Modules  1  and  2 
the  criterion  test  scores  in  this  hypothetical  example  would  indicate  that 
the  trainees  do  learn.  For  Module  3,  the  training  given  is  suspect.  Further 
analysis  is  required.  This  is  done  by  compiling  the  frequency  distribution 
of  errors  (missed  items)  on  the  Module  3  test  (Figure  7).  The  frequency 
distribution,  in  the  example  given,  shows  that  the  problem  area  is  concentrated 
in  test  items  7  through  13.  This  part  of  the  module,  both  content  and  test 
items,  should  be  analyzed  to  determine  what  the  problems  are  and  either 
cointent,  test  items  or  both  should  be  revised. 

In  addition  to  the  above  analysis  (and  assuming  trainees  do  learn  the 
stated  objectives,  the  relationship  of  trainee  characteristics  to  learning 
is  of  interest.  It  is  suggested  that  scatterplots  be  made  showing  the 
relationship  between  such  variables  as  GT,  age,  education,  etc.,  and  test 
score.  The  scatterplots  will  easily  show  if  such  a  relationship  exists. 

Because  of  the  relatively  small  number  of  trainees  (10),  the  results  can 
only  be  considered  indicative  rather  than  definitive.  They  do  represent  an 
opportunity  to  obtain  as  much  data  from  the  study  as  possible  with  relatively 
little  added  effort. 
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Figure  6.  Frequency  Distribution  of  Module  Test  Scores 

(Hypothetical) . 


Test  Item 
Number 


(Entries  are  number  of 
trainees  missing  each 
item) 
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Frequency  Distribution  of  Error  Scores  on  Test  Items 
for  Module  Test  3  (Hypothetical) . 


What  Attitudes  do  Trainees  Have  Toward  the  TACFIRE  AI  Courseware? 

Answer  to  this  question  are  obtained  by  compiling  and  analyzing  the 
responses  to  the  TACFIRE  AI  Debriefing  Questionnaire.  At  the  same  time, 
it  will  also  identify  particular  problems,  if  they  exist,  which  can  be 
analyzed  and  corrected. 

The  techniques  for  the  analysis  are  the  same  previously  stated.  A  summary 
sheet  is  prepared  and  frequency  distributions  of  the  responses  to  each  item 
are  prepared  along  with  the  mean  and  range  of  values  for  each  response. 

These  are  supplemented  by  the  responses  of  the  trainees  to  the  open  ended 
questions  and  the  additional  comments  they  make. 

If  the  responses  of  the  trainees  show  that  they  like  the  TACFIRE  AI  course  and 
have  little  or  no  difficulty  in  taking  the  course,  then  the  TACFIRE  AI  course 
can  be  considered  accepted  by  the  trainees,  of  particular  interest  would 
be  the  relation  between  liking  the  course  and  test  scores,  between  age 
_rd  liking  the  course,  length  of  service  and  liking  the  course,  etc.  These 
can  be  demonstrated  by  generating  scatterplots  showing  the  two  variables 
under  consideration.  Because  of  the  relatively  small  sample,  these  results 
can  only  be  considered  as  showing  a  possible  trend  rather  than  conclusive. 
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OTHER  CONSIDERATIONS 


The  effort  expended  on  the  evaluation  study  will  show  the  effectiveness  and 
acceptabi 1 i ty  of  the  TACFIRE  AI  courseware.  there  is  another  area  which  is 
of  high  interest  to  the  Army:  Skill  Qualification  Tests  (SQT).  Skill  Qualifi¬ 
cation  Tests  are  designed  to  evaluate  the  individual  skill  level  as  the 
individual  performs  in  the  operational  setting. 

It  is  recommended  that  the  trainees  who  take  the  TACFIRE  AI  course  have  the 
opportunity  to  exercise  their  skills  in  an  evaluative,  operational  setting. 

This  would  require  the  preparation  of  a  scenario  with  both  off-line  (voice) 
and  on-line  (FO  outputs)  which  would  require  the  trainee  to  perform  the  operations 
covered  by  the  course  objectives.  With  this  relatively  small  amount  of  added 
effort,  the  Army  can  obtain  some  valuable  data  regarding  the  effectiveness  of 
the  TACFIRE  AI  courseware  in  raising  the  skill  level  of  the  individual. 


APPENDIX  A 


TACFIRE  DEBRIEFING  QUESTIONNAIRE 


NAME  AND  GRADE 

UNIT  _ 

INTERVIEWER 


_  SSAN 

PHONE  _ 

DATE 


1.  What  did  you  think  of  the  TACFIRE  AI  course  that  you  have  Just  completed? 


2.  My  attitude  toward  the  course  was  that  I  .  .  . 

(  )  disliked  it  very  much 

<  )  disliked  it 

(  )  neither  liked  nor  disliked  it 
(  )  liked  it 

(  )  liked  it  very  much 

3.  Instructions  for  taking  the  course  were  .  .  . 

(  )  very  difficult  to  understand 

(  )  difficult  to  understand 

(  )  borderline 

<  )  easy  to  understand 

(  )  very  easy  to  understand 
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c 

4.  On  which  console  did  you  take  the  TACFIRE  AI  course? 


FM  Module: 

ACC 

_  VFMED  _ 

_ MIOD 

ATI  Module: 

, 

ACC 

_  VFMED  _ 

_ MIOD 

!  AFU  Module: 

ACC 

__  VFMED  _ 

_ MIOD 

SPRT  Module: 

ACC 

_  VFMED  _ 

_  MIOD 

SYS  Module: 

ACC 

VFMED  _ 

_ MIOD 

I  ' 

w 

i 


I 


5.  Did  you  have  any  problems  or  difficulties  in  using  the  console  or 
interacting  with  the  computer? 

(  )  yes  (  )  no 

(If  "yes"  to  item  5)  please  describe  your  most  serious  problem 
or  difficulty. 

j 

_  j 

6.  The  course  covered  five  functional  areas:  fire  missions,  artillery  ? 

target  intelligence,  ammunition  and  fire  unit,  support  and  system  i 

functions.  Were  any  of  these,  or  parts  of  these,  particularly  good,  i 

*  ' 

and  tell  why.  p  ! 


fi 


7 


Were  any  of  the  five  particularly  bad  and  tell  why 


8.  1  think  that  this  method  of  instruction/learning  is  .  .  . 


(  ) 

very  effective 

1 

t 

(  ) 

effective 

(  ) 

borderline 

(  ) 

ineffective 

(  ) 

very  ineffective 

\ 

For 

satisfactory  understanding  of  the  subject  being  studied, 

i 

1 

;  | 

the 

amount  of  time  provided  was: 

»  i 

•»  \ 

i 

(  ) 

much  too  long 

i 

(  ) 

fairly  long 

(  ) 

about  right 

(  ) 

fairly  short 

• 

s 

(  ) 

much  too  short 

1 

1 

i 

10.  For  satisfactory  understanding  of  the  subject  being  studied,  the 
amount  of  material  (information)  provided  was: 

(  )  much  too  large 

(  )  fairly  large 

(  )  about  right 

(  )  fairly  small 

(  )  much  too  small 

11.  The  technical  detail  provided  was: 

(  )  very  satisfactory 

(  )  satisfactory 

(  )  borderline 

(  )  unsatisfactory 

(  )  very  unsatisfactory 

12.  The  organization  of  the  material  presented  was: 

(  )  very  satisfactory 

(  )  satisfactory 

(  )  borderline 

(  )  unsatisfactory 

(  )  very  unsatisfactory 

13.  My  understanding  of  the  material  presented  was: 

(  )  very  satisfactory 

(  )  satisfactory 

(  )  borderline 

(  )  unsatisfactory 

(  )  very  unsatisfactory 
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14.  The  quantity  of  the  off-line  course  exhibits  provided  was: 


(  )  very  satisfactory 

(  )  satisfactory 

(  )  borderline 

(  )  unsatisfactory 

(  )  very  unsatisfactory 

15.  Were  any  of  the  off-line  course  exhibits  inaccurate? 
(  )  yes  (  )  no 

If  yes,  please  describe:  _ 


If,.  Were  any  of  the  off-line  course  exhibits  irrelevant  or  unnecessary? 
(  )  yes  (  )  no  If  yes,  which?  _ 


17.  Can  you  think  of  any  other  off-line  course  exhibits  that  should  be 
added  to  the  set? 

(  )  yes  (  )  no  If  yes,  please  describe:  _ 


18.  If  you  were  at  the  Artillery  Control  Console  (ACC)  now,  how  well  could 
you  take  the  console  actions  covered  in  the  course? 

(  )  very  effectively 

(  )  effectively 

(  )  borderline 

(  )  ineffectively 

(  )  very  ineffectively 
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19.  Have  you  now  learned  enough  about  ACC  operations  that  you  are  ready 
operate  it  in  the  areas  covered  by  the  course? 

(  )  yes  (  )  no  (  )  not  sure 

20.  If  "no"  or  "not  sure",  what  more  do  you  need  to  be  ready  to  operate 
the  ACC  in  these  areas? 


21.  Have  you  ever  had  this  kind  of  training  before? 
(  )  yes  (  )  no 


22.  Does  this  kind  of  training  make  Army  instruction  better? 


